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Abstract 



We report on a search for pair production of a fourth generation charge 
— 1/3 quark (b r ) in pp collisions at ^fs = 1.8 TeV at the Fermilab Tevatron 
using an integrated luminosity of 93 pb _1 . Both quarks are assumed to decay 
via flavor changing neutral currents (FCNC). The search uses the signatures 
7 + 3 jets +/i-tag and 27 + 2 jets. We see no significant excess of events over 
the expected background. We place an upper limit on the production cross 
section times branching fraction that is well below theoretical expectations 
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for a b' decaying exclusively via FCNC for b' masses up to mz + mi,. 
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The existence of three generations of quarks and leptons is well established in the Stan- 
dard Model. There is no strong expectation of additional quark and lepton generations in 
an extended Standard Model, nor are additional generations ruled out. Several models with 
new generations or arguments favoring new generations have been presented [0]. In this 
paper, we report on a search for pair production of a fourth generation charge —1/3 quark 
(b') that decays via flavor changing neutral currents (FCNC) in pp collisions at a/s =1.8 
TeV at the Fermilab Tevatron. While most Standard Model FCNC processes are highly 
suppressed, it is quite plausible that a light b' quark (i.e. my < m t and my < m t i) could 
decay predominantly via FCNC if, as expected, the charged current decay of a light b' quark 
to a charm quark is highly suppressed by a four-generation extension of the CKM matrix . 
The condition for FCNC dominance is roughly |l^&'/^t'&l < 10~ 2 to 10~ 3 depending on the 
mass of the b' and t' quarks. Several e + e~ collider experiments have explicitly searched for 
b' quarks decaying via FCNC |§, but until now there have been no searches for b' quarks 
that decay via FCNC at hadron colliders |3J. The current mass limit on a b' quark that 
decays via FCNC is the LEP I limit of half the Z boson mass || . 

The data used in this search were collected with the D0 detector during the 1992-1995 
Tevatron collider run and represent an integrated luminosity of 93 pb _1 . 

We assume that b' quarks are pair produced with the same cross section, for a given 
mass, as the top quark ||. We consider the signatures b'b' — > •ygbb and b'b' — > 7766, in 
which the photons are observed directly and b quarks and gluons are observed as hadronic 
jets. In the case of the single photon signature, we require that one of the b quark jets have 
a soft muon tag. We assume that b' quarks decay 100% of the time via FCNC with the 
relative FCNC branching fractions determined by the Standard Model [[!]], which are 13% 
for the single photon signature and 1.6% for the diphoton signature for a b' quark of mass 
80 GeV/c 2 . We have not included the three-body hadronic FCNC decay modes, such as 
b' — > bqq, in the single photon acceptance calculation or in any quoted theoretical branching 
fractions. The acceptance for such modes is only slightly lower than for the two-body decay 
b' — > bg. If three-body hadronic decay modes were included in the acceptance calculation 
for the single photon signature, the acceptance times branching fraction might increase by 
30-50%, but with considerable theoretical uncertainty. For b' masses above mz + m^, the 
decay channel b' — > Z + b is expected to dominate other FCNC decay processes. Thus, the 
sensitivity of the photon decay channels is limited to b' masses where the Z boson decay 
channel is not open. 

The D0 detector is described in detail in Ref. ||. The detector consists of an iron toroid 
muon spectrometer, a uranium-liquid argon calorimeter, and a non-magnetic central tracking 
volume containing drift chambers, a vertex chamber, and a transition radiation detector. 
Jets are reconstructed using a cone algorithm with radius TZ = 0.5 in i]-<f) space, where 
7] is pseudorapidity and <fi is the azimuthal angle. Muons are identified by reconstructed 
tracks in the muon spectrometer. Muons used for 6-tagging are required to have px > 4 
GeV/c, \r}\ < 1.1, and to be within ATI < 0.5 of a jet axis in rj-cj) space. Photon candidates 
are identified by the longitudinal and transverse shower shape of isolated calorimeter energy 
clusters, and by the absence of tracking chamber hits in the central tracking volume between 
the calorimeter cluster and the event vertex |J. The photon isolation requirement is that 
the energy in an annular isolation cone from radius 0.2 to 0.4 in f]-(p space be less than 10% 
of the photon energy. 
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In addition to requiring the requisite number of photons, jets, and 6-tagging muons, both 
analyses place a cut on the quantity Ht, which is defined as the scalar sum of the transverse 
energies (E T 's) of the photons, jets, and any 6-tagging muons in the event. Both analyses 
require H? > 1.6 ray. Note that the Ht cut depends explicitly on the b' mass hypothesis. 
The value of the Ht cut is set to maximize expected significance, defined as acceptance 
divided by the square root of the expected background. The cuts used by the two analyses 
are summarized in Table |I[ 

The acceptance is calculated using the herwig event generator [|K| with a detector 
simulation based on the geant program [JTTJ . The calculated acceptance for the two channels 
is listed in Tables [TT] and fTJ, respectively. 

We find 71 events before the Ht cut in the single photon analysis. The primary back- 
grounds to the single photon channel are QCD direct photon plus multijet production and 
QCD multijet production with one jet misidentified as a photon. Other backgrounds that 
are considered are Wj and Z7 production and W and Z bosons decaying to electron(s) 
with one electron misidentified as a photon. The sum of the direct photon and multijet 
backgrounds is calculated using the tag rate method, in which untagged 7 + 3 jet events are 
weighted by a per jet 6-tagging probability measured in multijet (> 4 jet) data. Figure [l](a) 
shows a test of the 6-tag rate in "bad 7" + 3 jets events, in which the photon has failed one 
of the photon identification cuts. There is good agreement between data and background. 
The estimated background obtained using the tag rate method is 62.8 ± 6.3 events before 
the Ht cut. The diboson background, which is expected to generate 6-tags in excess of the 
tag rate, is estimated by a Monte Carlo calculation to be 0.7 ± 0.4 events. The background 
from W and Z bosons decaying to electrons that are misidentified as photons is estimated 
to be 0.1 ±0.1 events and is not included in the subtracted background. The total expected 
background before the H T cut is 63.4 ± 6.3 events. The H T distributions of data and ex- 
pected background are shown in Fig. |l](b). There is a slight, but not statistically significant, 
excess of data over background. 

We find 20 events before the Ht cut in the diphoton channel. The primary backgrounds 
to the diphoton channel are QCD multijet production with two jets misidentified as pho- 
tons and single direct photon plus jets production with one jet misidentified as a photon. 
Other less important backgrounds are double direct photon + jets production and Z — > ee 
events where both electrons are misidentified as photons. The sum of the two fake photon 
backgrounds is estimated from the measured probability for a jet to be misidentified as the 
second photon in single photon candidate plus three jet events, corrected for the fraction of 
photon candidates that are actually photons (the photon purity) of the first photon. The 
purity of the first photon in the 27 + 2 jet sample is estimated to be 38 ± 16% |J. 

The sum of the two fake backgrounds is estimated to be 14.5 ± 2.2 events before the Ht 
cut. The double direct photon background is estimated by Monte Carlo calculation to be 
1.2 ± 0.6 events. The Z — > ee background is estimated to be 0.1 ± 0.1 events and is not 
included in the subtracted background. The total expected background before the H T cut is 
15.7±2.3 events. There is again a slight, but not statistically significant, excess of data over 
background. The Ht distributions of data and expected background are shown in Fig. |l](c). 

The acceptance, the number of data events, the expected signal and background, in- 
cluding the effect of the variable Ht cut, and the calculated cross section times branching 
fraction are shown in Tables [TI| and [01]. The 95% confidence level (CL) upper limit on 
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the cross section times branching fraction is calculated using Gaussian errors excluding the 
unphysical negative cross section region. Using the theoretical production cross section of 
Laenen et al. 0, including the quoted theoretical uncertainty, we derive an upper limit on 
the branching fraction for each of the two channels. The 95% confidence level upper limit for 
both channels is shown in Figs. |2](a) and (b). Using the theoretical relative FCNC branching 
fractions of Ref . , we derive an upper limit on the total FCNC branching fraction of the 
b' quark for both channels individually and combined. The combined upper limit on the 
FCNC branching fraction of the b' quark is shown in Fig. |](c). For both channels the upper 
limit on the branching fraction is well below the theoretical branching fraction for a b' quark 
that decays 100% of the time via FCNC. The upper limit on the total FCNC branching 
fraction of the b' quark is less than 50%, for all masses up to mz + rrib, at which point the 
Z boson decay channel opens up. 

We thank the staffs at Fermilab and the collaborating institutions for their contribu- 
tions to the success of this work, and acknowledge support from the Department of Energy 
and National Science Foundation (U.S.A.), Commissariat a L'Energie Atomique (France), 
Ministries for Atomic Energy and Science and Technology Policy (Russia), CNPq (Brazil), 
Departments of Atomic Energy and Science and Education (India), Colciencias (Colombia), 
CONACyT (Mexico), Ministry of Education and KOSEF (Korea), CONICET and UBACyT 
(Argentina), and the A. P. Sloan Foundation. 
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FIG. 1. Ht distributions of data (filled squares) and expected background (open circles) in the 
channels (a) "bad 7" + 3 jets, (b) 7 + 3 jets, and (c) 27 + 2 jets. 
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FIG. 2. Measured 95% confidence level upper limit on the branching fraction (solid line) and 
theoretical branching fraction (dotted line) for (a) b'b' -^7 + 3 jets, (b) b'b' — ► 2j + 2 jets, and (c) 
the total FCNC branching fraction of b' . The theoretical and FCNC branching fraction curves end 
at my = mz + mj, due to the opening of the Z boson FCNC decay channel. 
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TABLES 



TABLE I. Kinematic cuts used in the 7 + 3 jets and 27 + 2 jets analyses (energies in GeV). 





Photons 


Jets 






Channel 


N 


Ermin 


I f] | max 


A^iiiii 


ETmin 


I V 1 max 


6-tag 


ETmin 


7 + 3j 


1 


20 


1 


3 


15 


2 


yes 


1.6 my 


2 7 + 2j 


2 


20 


2 


2 


15 


2.5 


no 


1.6 my 



TABLE II. The acceptance, the numbers of expected and observed events, and the measured 
cross section as a function of b' mass in the 7 + 3 jets channel. The acceptance includes the muon 
semileptonic branching fraction of the b quark. The integrated luminosity is 93 pb~L 



my 
(GeV/c 2 ) 


Acceptance 
(%) 


Events 


o vv x B(b'b' 


-> jgbb) (pb) 


Observed 


Expected 
Signal (B = 13%) 


Expected 
Background 


Value 


Upper limit 
(95% CL) 


50 


0.38 ± 0.07 


71 


166 ± 33 


63.1 + 6.3 


22.1 + 30.1 


75.3 


60 


0.64 + 0.12 


70 


115 + 22 


60.0 + 6.0 


17.0 + 17.8 


47.8 


70 


1.10 + 0.19 


60 


87 + 16 


53.4 + 5.3 


6.5 + 9.3 


23.1 


80 


1.45 + 0.25 


46 


57 + 10 


45.4 + 4.6 


0.4 + 6.1 


12.2 


90 


1.68 + 0.29 


30 


35 + 6 


37.4 + 3.8 


-4.8 + 4.4 


6.0 


100 


2.16 + 0.36 


23 


26 + 5 


30.1 + 3.1 


-3.5 + 2.9 


3.9 


120 


2.88 ± 0.46 


14 


13 + 2 


18.7 + 1.9 


-1.8 + 1.6 


2.2 


140 


3.50 + 0.55 


9 


7+1 


12.0 + 1.3 


-1.0 + 1.0 


1.5 



TABLE III. The acceptance, the numbers of expected and observed events, and the measured 
cross section as a function of b' mass in the 27 + 2 jets channel. The integrated luminosity is 79 
pb" 1 . 







Events 


Gyy x B(b'b' 


-> 7766) (pb) 


my 


Acceptance 




Expected 


Expected 




Upper limit 


(GeV/c 2 ) 


(%) 


Observed 


Signal (B = 1.6%) 


Background 


Value 


(95% CL) 


50 


2.76 ± 0.40 


20 


126 ± 20 


15.5 + 2.3 


2.03 + 2.33 


6.11 


60 


5.31 + 0.71 


18 


101 ± 15 


14.1 + 2.1 


0.91 ± 1.14 


2.91 


70 


8.19 + 1.08 


15 


68.3 + 9.7 


11.0 + 1.7 


0.61 + 0.66 


1.76 


80 


10.45 ± 1.37 


11 


42.9 + 6.1 


8.4 + 1.3 


0.31 + 0.44 


1.08 


90 


11.90 ± 1.52 


8 


26.3 + 3.6 


6.2 ± 1.0 


0.18 + 0.32 


0.76 


100 


13.23 ± 1.68 


6 


16.5 + 2.3 


4.4 + 0.8 


0.15 + 0.25 


0.59 


120 


15.73 ± 2.00 


3 


7.5 + 1.0 


2.4 + 0.5 


0.05 + 0.15 


0.32 


140 


16.28 + 2.06 


3 


3.4 + 0.5 


1.5 + 0.4 


0.11 + 0.14 


0.36 
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